Electronic imaging cameras for recording still images are well known in the art. Such electronic imaging cameras may utilize two-dimensional electronic image sensing arrays such as charge-couple devices (CCD's) or complementary metal oxide semiconductor (CMOS) devices. Such two-dimensional image sensing devices typically comprise a regular array of image-sensing elements (cf., pixels) that respond to incident illumination and provide an electronic signal corresponding to the intensity of this illumination. The electronic signals from the image sensing elements collectively form image data, which may be recorded in either analog or digital form. The signals may also, either immediately or subsequently, be displayed on viewing devices such as cathode-ray tubes or liquid crystal displays, or be printed to provide a hard copy.
The image sensing elements, on their own accord, discriminate primarily the intensity of the incident light, and not its color. In order to record a color image, it is common to cover each element with a color filter chosen from a small collection of filter colors.
Typical color imaging devices use one of two-color schemes. The first color scheme uses red, green and blue filters. The second color scheme uses cyan, magenta and yellow filters. In this way, each image-sensing element is made sensitive to the intensity of light having the color of its overlying filter. For example, an element covered by a red filter is responsive to red light, an element with a green filter to green light, and so on.
The disposition of the color filters on top of the array of sensing elements can be one of several well-known patterns. One such disposition is a repeating pattern of red, green and blue vertical stripes, so that all sensing elements within a single column respond to the intensity of a single color of light. Other dispositions include checkerboard arrays.
In such arrays, it is common practice to use a repeating pattern of colors. For example, one well-known and popular pattern of filters is known as the Bayer pattern. In the Bayer pattern, even numbered rows of the image-sensing device have alternating red and green filters, while odd numbered rows have alternating green and blue filters (the blue filters aligned with the green filters of the even numbered rows).
This method of collecting color image data, although widely used, suffers from the problem that each image sensing element records the intensity of only a single color. (i.e., each image sensing element generates one-color image data) Image printing and display devices, on the other hand, commonly require information about the intensity of all of the colors at every pixel. The remedy to this problem is a process called “color recovery”, in which the missing color intensities at each of the pixels are estimated from the data at surrounding pixels.
Conventionally, the missing color data is filled in by 2-dimensional interpolation. For example, when a Bayer pattern is used, each blue-sensing element is surrounded by four green-sensing elements, and the missing value of green at the blue-sensing element may be estimated as the average of the data from the four green-sensing neighbors.
Conventional color recovery methods can produce images with objectionable artifacts, such as “color fringes”, near sharp edges. The usual approach to solving this problem eliminates color fringes at the expense of image sharpness either by blurring the picture or by suppressing selected spatial frequencies, known as anti-aliasing. Treating the image in either manner has the disadvantage of producing a blurred image.
A solution to the artifact problem is described by W. T. Freeman in U.S. Pat. Nos. 4,663,655 and 4,774,565, both of which are herein incorporated. These patents set forth a color recovery scheme that uses nonlinear-filtering to produce an image that is sharp, correctly colored, and having a reduced incidence of the aforementioned artifacts.
In the Freeman patents, a color-recovery process and an apparatus for the implementation thereof are disclosed. The process could be applied either on 1-dimensional data, produced by a single-line array of sensing elements with color filters, such as in a flat-bed image scanner, or on 2-dimensional data, produced by a 2-dimensional array of sensing elements with color filters, such as in an electronic camera. The method is easiest to describe in the case of a 1-dimensional linear array having a repeating pattern of red-, green- and blue-sensing elements i. e., RGBRGB etc.
First, in accordance with said method, for each color pixel type (e.g., red), there will be one actually measured value for that color at every third sensing element, with values for that color missing at the two intervening pixels. The first step thus is to fill in the missing red (R) values at green (G) and blue (B) elements by using any of a number of well-known interpolation methods.
Secondly and similarly, the blue and green values are filled in for all pixels at which they have not been explicitly measured, using the same interpolation method.
Third, at this point red (R), green (G) and blue (B) values exist for every pixel. One of these values is an explicitly measured value, and the other two have been obtained by interpolation. The interpolated values are denoted by attaching a prime, as in R′ or G′ or B′. The interpolated data now have the form:
Element 1Element 2Element 3Element 4Element 5 . . .R, G′, B′R′, G, B′R′, G′, BR, G′, B′R′, G, B′
Fourth, at each pixel, the three color difference values, R−G, G−B, and B−R are calculated using either explicitly measured values or interpolated values such as they are available at each pixel.
Fifth, the R−G values across the row are then collected and subjected to a median filter of some predetermined length, typically an odd number in the range 5–11. The same is done with the G−B values and the B−R values, so that at each pixel there is a complete set of median-filtered color-difference values. These are denoted as (R−G)mf, (G−B)mf and (B−R)mf.
Sixth, at each pixel, the one explicitly measured color is combined with the two median-filtered color difference signals that contain that color to produce improved estimates for the previously interpolated color values. For the example array described above, the first pixel has an explicitly measured value for R, but interpolated values for G′ and B′. In that case, the R values are combined with the median filtered color differences (R−G)mf, and (B−R)mf to find the final estimates:G^=R−(R−G)mfB^=R+(B−R)mfwhere the ^ notation is used to denote a final recovered data value. This prescription is followed for each of the pixels, producing final color estimates that have the desirable properties claimed in the Freeman patents.
The Freeman method may be practiced on two-dimensional color data as well, using essentially the same prescription. In that case, for each sensing element, calculations are performed on a symmetrically disposed neighborhood of elements. In steps 1 and 2, standard 2-dimensional interpolation methods, such as bi-cubic interpolation, are used to produce the initial guesses for the missing colors at the element.
Then, all of the color differences are found at each element, and in step 5, a 2-dimensional median filter is applied to the neighborhood of the element to find the median-filtered color difference values.
Finally, in step 6, improved estimates for the colors are calculated for colors that have not been explicitly measured at each pixel. This is done by combining the one color that has been measured with all of the median-filtered color-difference values that contain that color.
It should be appreciated that in each case, any missing color is finally determined by combining a measured color value with a single median-filtered color difference.
While the methods described in the Freeman patent provide desirable results in certain applications, when practiced on comparatively large two-dimensional color data arrays, the number of calculations needed for complete color recovery becomes correspondingly large. Since the speed of the methodology is tied to the number of its required calculation, there is an existing need for a method for recovering missing color data in a two-dimensional color data array that provides results comparable to those obtainable through the Freeman methodologies, yet not requiring as many calculations.